Procuring feedback

ABSTRACT

A system for procuring feedback from user when displaying content. The system implements operations for selecting a feedback mechanism based on prior user interactions with the feedback mechanism and other feedback mechanisms. The system uses the user responses to the feedback mechanism to better target content to the user and better determine which feedback mechanisms will provide the most valuable information over time.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/193,591, filed Jun. 27, 2016, which is herein incorporated byreference in its entirety.

TECHNICAL FIELD

The present disclosure generally relates to encouraging user feedback,and more specifically, providing feedback options that tailor contentfor a user and tailor feedback mechanisms based on the userinteractions.

BACKGROUND

Obtaining user feedback on applications is difficult. Developers oftenfind it difficult to get user engagement in providing active feedback.Furthermore, a one size fits all feedback model is more likely to get nofeedback or a small subset of users to provide feedback. Thus it wouldbe beneficial if there were a dynamically changing feedback mechanismthat is personalized to each user.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram of an example computing system that adaptedfor encouraging user feedback to content.

FIG. 2 is a block diagram of an example computer system suitable forimplementing one or more devices of the computing system in FIG. 1.

FIG. 3 is a flow diagram illustrating an example dynamic feedbackprocess.

FIG. 4 illustrates an example graphical user interface of a feedbackmechanism having selectable keywords.

FIG. 5 illustrates an example graphical user interface of a feedbackmechanism that implements a slider for feedback.

Embodiments of the present disclosure and their advantages are bestunderstood by referring to the detailed description that follows. Itshould be appreciated that like reference numerals are used to identifylike elements illustrated in one or more of the figures, whereasshowings therein are for purposes of illustrating embodiments of thepresent disclosure and not for purposes of limiting the same.

DETAILED DESCRIPTION

In the following description, specific details are set forth describingsome embodiments consistent with the present disclosure. It will beapparent, however, to one skilled in the art that some embodiments maybe practiced without some or all of these specific details. The specificembodiments disclosed herein are meant to be illustrative but notlimiting. One skilled in the art may realize other elements that,although not specifically described here, are within the scope and thespirit of this disclosure. In addition, to avoid unnecessary repetition,one or more features shown and described in association with oneembodiment may be incorporated into other embodiments unlessspecifically described otherwise or if the one or more features wouldmake an embodiment non-functional.

In some embodiments of the present disclosure, systems and methods forobtaining feedback for targeted content is disclosed. In some examples,the system may have several different feedback requesting options toselect from. The system, in some examples, may determine the bestfeedback requesting option to provide to a user with targeted contentbased on prior user interactions with the different feedback requestingoptions and/or other user interactions with the feedback requestingoptions. In some examples, response to the feedback requests from usersmay be used to tailor how content is targeted to the user. In someexamples, the system may aggregate and orchestrate the collection ofuser data with and/or on behalf of third-party entities to maximizeand/or increase user engagement with content.

In some embodiments a feedback system and method is provided. In someexamples the system includes a non-transitory memory storinginstructions and one or more hardware processors coupled to thenon-transitory memory and configured to read the instructions from thenon-transitory memory to cause the system to perform operations. Theoperations may include selecting an advertisement based at least in parton an advertisement keyword matching a profile keyword associated withuser information identified by a received user identifier, causing auser device associated with the received user identifier to display theadvertisement with a feedback request, the feedback request having anactuatable keyword that matches the advertisement keyword, receiving,from the user device, an indication as to whether the actuatable keywordwas actuated, and adjusting the user information based on whether theactuatable keyword was actuated.

In some examples, a computer implemented method is disclosed. The methodmay include operations such as selecting an advertisement based at leastin part on an advertisement keyword matching a profile keywordassociated with user information identified by a received useridentifier, causing a user device associated with the received useridentifier to display the advertisement with a feedback request, thefeedback request having a virtual slider associated with theadvertisement keyword, receiving, an indication that the virtual sliderwas moved; and adjusting the user information based on receiving theindication that the virtual slider was moved.

Some embodiments disclose a non-transitory machine readable mediumhaving stored thereon machine readable instructions executable to causea machine to perform operations. In some examples the operation mayinclude choosing an advertisement based at least in part on a contentkeyword matching a profile keyword associated with user informationidentified by a received user identifier, causing a mobile deviceassociated with the received user identifier to display the content witha feedback request, receiving a response to the feedback request, andadjusting the user information based on receiving the response to thefeedback request.

FIG. 1 illustrates, in block diagram format, an example embodiment of acomputing system adapted for encouraging use feedback to content. Asshown, a computing system 100 may comprise or implement a plurality ofservers and/or software components that operate to perform variousmethodologies in accordance with the described embodiments. Some exampleservers may include, for example, stand-alone and enterprise-classservers operating a server operating system (OS) such as a MICROSOFT®OS, a UNIX® OS, a LINUX® OS, or other suitable server-based OS. It maybe appreciated that the servers illustrated in FIG. 1 may be deployed inother ways and that the operations performed and/or the servicesprovided by such servers may be combined, distributed, and/or separatedfor a given implementation and may be performed by a greater number orfewer number of servers. One or more servers may be operated and/ormaintained by the same or different entities.

Computing system 100 may include, among various devices, servers,databases and other elements, one or more clients 102 that may compriseor employ one or more client devices 104, such as a laptop, a mobilecomputing device, a tablet, a PC, a wearable device, and/or any othercomputing device having computing and/or communications capabilities inaccordance with the described embodiments. Client devices 104 mayinclude a cellular telephone, smart phone, electronic wearable device(e.g., smart watch, virtual reality headset), or other similar mobiledevices that a user may carry on or about his or her person and accessreadily.

Client devices 104 generally may provide one or more client programs106, such as system programs and application programs to perform variouscomputing and/or communications operations. In some examples, systemprograms may include, without limitation, an operating system (e.g.,MICROSOFT® OS, UNIX® OS, LINUX® OS, Symbian OS™, Embedix OS, BinaryRun-time Environment for Wireless (BREW) OS, JavaOS, a WirelessApplication Protocol (WAP) OS, and others), device drivers, programmingtools, utility programs, software libraries, application programminginterfaces (APIs), and so forth. In some examples, application programsmay include, without limitation, a web browser application, messagingapplications (e.g., e-mail, IM, SMS, MMS, telephone, voicemail, VoIP,video messaging, internet relay chat (IRC)), contacts application,calendar application, electronic document application, databaseapplication, media application (e.g., music, video, television),location-based services (LBS) applications (e.g., GPS, mapping,directions, positioning systems, geolocation, point-of-interest,locator) that may utilize hardware components such as an antenna, and soforth. One or more of client programs 106 may display various graphicaluser interfaces (GUIs) to present information to and/or receiveinformation from one or more users of client devices 104. In someembodiments, client programs 106 may include one or more applicationsconfigured to conduct some or all of the functionalities and/orprocesses discussed below.

As shown, client devices 104 may be communicatively coupled via one ormore networks 108 to a network-based system 110. Network-based system110 may be structured, arranged, and/or configured to allow client 102to establish one or more communications sessions between network-basedsystem 110 and various computing devices 104 and/or client programs 106.Accordingly, a communications session between client devices 104 andnetwork-based system 110 may involve the unidirectional and/orbidirectional exchange of information and may occur over one or moretypes of networks 108 depending on the mode of communication. While theembodiment of FIG. 1 illustrates a computing system 100 deployed in aclient-server operating environment, it is to be understood that othersuitable operating environments and/or architectures may be used inaccordance with the described embodiments.

Data communications between client devices 104 and the network-basedsystem 110 may be sent and received over one or more networks 108 suchas the Internet, a WAN, a WWAN, a WLAN, a mobile telephone network, alandline telephone network, personal area network, as well as othersuitable networks. For example, client devices 104 may communicate withnetwork-based system 110 over the Internet or other suitable WAN bysending and or receiving information via interaction with a web site,e-mail, IM session, and/or video messaging session. Any of a widevariety of suitable communication types between client devices 104 andsystem 110 may take place, as will be readily appreciated. Inparticular, wireless communications of any suitable form may take placebetween client device 104 and system 110, such as that which oftenoccurs in the case of mobile phones or other personal and/or mobiledevices.

In various embodiments, computing system 100 may include, among otherelements, a third party 112, which may comprise or employ third-partydevices 114 hosting third-party applications 116. In variousimplementations, third-party devices 114 and/or third-party applications116 may host applications associated with or employed by a third party112. For example, third-party devices 114 and/or third-partyapplications 116 may enable network-based system 110 to provide client102 and/or system 110 with additional services and/or information, suchas merchant information, data communications, payment services, securityfunctions, customer support, and/or other services, some of which willbe discussed in greater detail below. Third-party devices 114 and/orthird-party applications 116 may also provide system 110 and/or client102 with other information and/or services, such as email servicesand/or information, property transfer and/or handling, purchase servicesand/or information, and/or other online services and/or information.

In one embodiment, third-party devices 114 may include one or moreservers, such as a content server that provides content for display. Insome embodiments, the third-party devices may include a user and/or userdevice database that can provide information and profile informationgathered from users and/or user devices by third party 112. In yetanother embodiment, third-party severs 114 may include one or moreservers for aggregating user data, content data, and other statistics.

Network-based system 110 may comprise one or more communications servers120 to provide suitable interfaces that enable communication usingvarious modes of communication and/or via one or more networks 108.Communications servers 120 may include a web server 122, an API server124, and/or a messaging server 126 to provide interfaces to one or moreapplication servers 130. Application servers 130 of network-based system110 may be structured, arranged, and/or configured to provide variousonline services, account management, advertisement handling, contenthandling, data gathering, data analysis, and other services to usersand/or third parties that access network-based system 110. In variousembodiments, client devices 104 and/or third-party devices 114 maycommunicate with application servers 130 of network-based system 110 viaone or more of a web interface provided by web server 122, aprogrammatic interface provided by API server 124, and/or a messaginginterface provided by messaging server 126. It may be appreciated thatweb server 122, API server 124, and messaging server 126 may bestructured, arranged, and/or configured to communicate with varioustypes of client devices 104, third-party devices 114, third-partyapplications 116, and/or client programs 106 and may interoperate witheach other in some implementations.

Web server 122 may be arranged to communicate with web clients and/orapplications such as a web browser, web browser toolbar, desktop widget,mobile widget, web-based application, web-based interpreter, virtualmachine, mobile applications, and so forth. API server 124 may bearranged to communicate with various client programs 106 and/or athird-party application 116 comprising an implementation of API fornetwork-based system 110. Messaging server 126 may be arranged tocommunicate with various messaging clients and/or applications such ase-mail, IM, SMS, MMS, telephone, VoIP, video messaging, IRC, and soforth, and messaging server 126 may provide a messaging interface toenable access by client 102 and/or third party 112 to the variousservices and functions provided by application servers 130.

Application servers 130 of network-based system 110 may be a server thatprovides various services to clients and or third parties including, butnot limited to, content targeting, advertisement targeting, contentfeedback handling, advertisement selection handling, user profiling,and/or the like. Application servers 130 may include an account server132, device identification server 134, content server 136, contentselection server 138, profile merging server 140, user identificationserver 142, feedback server 144, and/or content statistics server 146.These servers, which may be in addition to other servers, may bestructured and arranged to configure the system for providing contentand obtaining feedback associated with the content.

Application servers 130, in turn, may be coupled to and capable ofaccessing one or more databases 150 including a profile database 152, acontent database 154, and/or feedback database 156. Databases 150generally may store and maintain various types of information for use byapplication servers 130 and may comprise or be implemented by varioustypes of computer storage devices (e.g., servers, memory) and/ordatabase structures (e.g., relational, object-oriented, hierarchical,dimensional, network) in accordance with the described embodiments.

FIG. 2 illustrates an example computer system 200 in block diagramformat suitable for implementing on one or more devices of the computingsystem in FIG. 1. In various implementations, a device that includescomputer system 200 may comprise a personal computing device (e.g., asmart or mobile phone, a computing tablet, a personal computer, laptop,wearable device, PDA, etc.) that is capable of communicating with anetwork. A service provider and/or a content provider may utilize anetwork computing device (e.g., a network server) capable ofcommunicating with the network. It should be appreciated that each ofthe devices utilized by users, service providers, and content providersmay be implemented as computer system 200 in a manner as follows.

Additionally, as more and more devices become communication capable,such as new smart devices using wireless communication to report, track,message, relay information and so forth, these devices may be part ofcomputer system 200. For example, windows, walls, and other objects maydouble as touch screen devices for users to interact with. Such devicesmay be incorporated with the systems discussed herein.

Computer system 200 may include a bus 202 or other communicationmechanisms for communicating information data, signals, and informationbetween various components of computer system 200. Components include aninput/output (I/O) component 204 that processes a user action, such asselecting keys from a keypad/keyboard, selecting one or more buttons,links, actuatable elements, etc., and sends a corresponding signal tobus 202. I/O component 204 may also include an output component, such asa display 211 and a cursor control 213 (such as a keyboard, keypad,mouse, touchscreen, etc.). In some examples, I/O component 204 mayinclude an image sensor for capturing images and/or video, such as acomplementary metal oxide semiconductor (CMOS) image sensor, and/or thelike. An audio input/output component 205 may also be included to allowa user to use voice for inputting information by converting audiosignals. Audio I/O component 205 may allow the user to hear audio. Atransceiver or network interface 206 transmits and receives signalsbetween computer system 200 and other devices, such as another userdevice, a merchant server, an email server, application serviceprovider, web server, a payment provider server, and/or other serversvia a network. In various embodiments, such as for many cellulartelephone and other mobile device embodiments, this transmission may bewireless, although other transmission mediums and methods may also besuitable. A processor 212, which may be a micro-controller, digitalsignal processor (DSP), or other processing component, processes thesevarious signals, such as for display on computer system 200 ortransmission to other devices over a network 260 via a communicationlink 218. Again, communication link 218 may be a wireless communicationin some embodiments. Processor 212 may also control transmission ofinformation, such as cookies, IP addresses, images, and/or the like toother devices.

Components of computer system 200 also include a system memory component214 (e.g., RAM), a static storage component 216 (e.g., ROM), and/or adisk drive 217. Computer system 200 performs specific operations byprocessor 212 and other components by executing one or more sequences ofinstructions contained in system memory component 214. Logic may beencoded in a computer readable medium, which may refer to any mediumthat participates in providing instructions to processor 212 forexecution. Such a medium may take many forms, including but not limitedto, non-volatile media, volatile media, and/or transmission media. Invarious implementations, non-volatile media includes optical or magneticdisks, volatile media includes dynamic memory such as system memorycomponent 214, and transmission media includes coaxial cables, copperwire, and fiber optics, including wires that comprise bus 202. In oneembodiment, the logic is encoded in a non-transitory machine-readablemedium. In one example, transmission media may take the form of acousticor light waves, such as those generated during radio wave, optical, andinfrared data communications.

Some common forms of computer readable media include, for example,floppy disk, flexible disk, hard disk, magnetic tape, any other magneticmedium, CD-ROM, any other optical medium, punch cards, paper tape, anyother physical medium with patterns of holes, RAM, PROM, EPROM,FLASH-EPROM, any other memory chip or cartridge, or any other mediumfrom which a computer is adapted to read.

In various embodiments of the present disclosure, execution ofinstruction sequences to practice the present disclosure may beperformed by computer system 200. In various other embodiments of thepresent disclosure, a plurality of computer systems 200 coupled bycommunication link 218 to the network (e.g., such as a LAN, WLAN, PTSN,and/or various other wired or wireless networks, includingtelecommunications, mobile, and cellular phone networks) may performinstruction sequences to practice the present disclosure in coordinationwith one another. Modules described herein may be embodied in one ormore computer readable media or be in communication with one or moreprocessors to execute or process the techniques and algorithms describedherein.

A computer system may transmit and receive messages, data, informationand instructions, including one or more programs (i.e., applicationcode) through a communication link and a communication interface.Received program code may be executed by a processor as received and/orstored in a disk drive component or some other non-volatile storagecomponent for execution.

Where applicable, various embodiments provided by the present disclosuremay be implemented using hardware, software, or combinations of hardwareand software. Also, where applicable, the various hardware componentsand/or software components set forth herein may be combined intocomposite components comprising software, hardware, and/or both withoutdeparting from the spirit of the present disclosure. Where applicable,the various hardware components and/or software components set forthherein may be separated into sub-components comprising software,hardware, or both without departing from the scope of the presentdisclosure. In addition, where applicable, it is contemplated thatsoftware components may be implemented as hardware components andvice-versa.

Software, in accordance with the present disclosure, such as programcode and/or data, may be stored on one or more computer readable media.It is also contemplated that software identified herein may beimplemented using one or more computers and/or computer systems,networked and/or otherwise. Such software may be stored and/or used atone or more locations along or throughout the system, at client 102,network-based system 110, or both. Where applicable, the ordering ofvarious steps described herein may be changed, combined into compositesteps, and/or separated into sub-steps to provide features describedherein.

The foregoing networks, systems, devices, and numerous variationsthereof may be used to implement one or more services, such as theservices discussed above and in more detail below.

FIG. 3 illustrates example dynamic feedback process 300 that may beimplemented by a system, such as system 100 of FIG. 1, according to someembodiments. According to some embodiments, process 300 may include oneor more of operations 301-308, which may be implemented, at least inpart, in the form of executable code stored on a non-transitory,tangible, machine-readable media that, when run on one or moreprocessors, may cause one or more systems to perform one or more of theoperations 301-308.

In some examples, process 300 may include operation 301. At operation301, the system may receive user information such as an identifier of auser for content targeting. In some examples the content may be anadvertisement, video content, a news article, virtual realityinteractive, mixed reality interactive, augmented reality interactive,videogame and/or the like. User information may include a useridentification, webpage browsing information, browsing session time,date, browsing session identifier, registration identifier, IP address,MAC address, device identifier, clickstream data, click path data,and/or other browsing information.

In some examples, the user information may be part of a web cookie orhypertext transfer protocol (HTTP) cookie. In some examples, the systemor a third-party system may store a cookie on the user device, and thecookie may be accessed by the system or third-party system to retrieveuser information. In cases where a third-party system retrieves the userinformation from the cookie the system may receive the user informationfrom the third-party system. In some examples, the user information maybe from several different third-party systems in conjunction withinformation retrieved by the system. For example, the system or one ormore third-party systems may obtain the IP address or MAC address of auser device through direct network communications with the user deviceand the protocol that is used to conduct the network communications. Inaddition the system and the one or more third-party systems may alsoobtain other information through one or more cookies that the system andone or more third-party systems have access to on the user device. Usingthe IP address, MAC address and/or other identifying information, thesystem may be able to determine that certain information from differentsources are associated with a single user and/or user device.

In some examples, user information may be provided by the user as partof an account creation. In some examples, user information may becollected by the system or one or more third-party systems and the useruses and application or website controlled by the system and/or the oneor more third parties. For example, the system and/or one or morethird-party systems may maintain and/or obtain user information and/orbrowsing information associated with a user as the user uses a websiteor application. For example, user information may be collected throughsocial media website or application controlled by the system and/orthird-party system. Social media websites generally have a login processwhich allows for the system or a third-party system to identify theuser.

In some examples, process 300 may include operation 302. At operation302, the system may retrieve and/or create a user profile. In someexamples, the system, using the information received at operation 301,may determine whether a user profile exists for the user. In someexamples, the system may determine if any of the received userinformation has identifying information that matches information for auser profile. For example, the system may check whether the IP address,MAC address, user ID, cookie ID, and/or other information received inoperation 301 matches the same information of any user profiles held ina database maintained, operated, and/or accessed by the system. In theevent that there is a match, the system may update and/or retrieve userinformation stored on the database that has the matching identificationinformation. In the event the system does not find a match, the systemmay create a new profile and update it with some or all of theinformation receive at operation 301.

In some examples, the system may aggregate information about a user frommultiple data collection systems, some which may be by a third-party. Inthis manner, content targeting may be based on a larger and morecomplete data set associated with a user and/or user device. In someexamples, the system may correlate data from different sources to createa user profile. In some examples, the data may be correlated based onsimilar user information elements such as IP addresses, MAC addresses,device addresses, user ID information, and/or the like. In someexamples, the system may determine several devices are associated with auser and attempt to merge and/or aggregate the user data associated withthe several devices. For example, a user device may conduct an actionthrough content provided by the system which may include the userproviding user identifying information, such as a name, address, emailaddress, and/or the like. When the user conducts another action withanother device, the system may determine that the device is associatedto the same user based on the name, address, email login information,and/or other information.

In some examples, the user profile may include browsing informationassociated with the user. For example, how websites were browsed, whatcontent s have been seen or clicked on, what information is associatedwith the website that the user browsed. In some examples, the userprofiles may include other information, such as what movies have beenwatched, what games have been played, and/or other information relatedto the different types of content. In some examples, the system may havereceive and/or the system may identify keywords associated with thecontent that the user browsed. In some examples, the system may collectthe keywords or be provided the keywords from a third-party system. Insome examples, the system may scrape the keywords from the content orcontent providers and store it as part of the user profile. The keywordsmay be obtained from different types of media sources such as audio,video, images, and/or other types of media. The system and/or a thirdparty system may convert media information into keywords by convertingaudio into text and/or conducting text recognition on images, videocontent, and/or the like.

Some of these operations may occur when a new profile is made and/orwhen a profile is updated. The system may leverage a natural languageprocessors to identify the keywords, such as identifying nouns, nounphrases, names of objects, and/or the like. The system may leverage adictionary of words to find keywords through keyword matching. Thedictionary may have brand names, merchandise names, location names,keywords associated with content, advertisements, subject categories,and/or the like. In some examples, the keywords for the dictionary maybe provided to the system from content providers.

In some examples, the user profile may include user preferences thathave been provided to the system, content, such as advertisements, thathave been targeted to the user, interactions the user may have had withcontent, when content was last targeted to the user (e.g. timestamp),and/or other information. In some examples, the system may maintain aledger or log of content metrics associated with a particular userand/or user device. For example, the content metrics may beadvertisement metrics which may include, but is not limited to, clicknumbers, impressions, actions (such as a resulting purchase throughadvertisement clicks), and/or the like.

In some examples, process 300 may include operation 303. At operation303, the system may select content to be directed to the user. In someexamples, such as when a new profile was created at operation 302, thecontent may be chosen based on the information received at operation301. In some examples, when a profile already exists, the system may useinformation associated with the profile to select content for use. Insome examples, the system may count the number of keywords that isassociated with the content that also matches keywords associated withthe profile. The different selectable content may be ranked from havingthe most matched keywords to the least. In some embodiments, thekeywords may have a value associated with the keywords and the systemmay sum the values of the keywords that are matching between aparticular content option and a user profile. In some examples, thevalues of each keyword may be specific to a user profile, to thecontent, or both. The different content may be ranked based on thehighest or lowest summed value of the matching keywords. The value ofthe keywords may be preset and/or change based on user feedback. Thesystem may select one of the content options that are highly ranked.

In some examples, the system may selected the highest ranked contentoption. In some examples, the system may select randomly from apredetermined number of the highest ranked content options. In someexamples, the system may select randomly from all of the ranked contentoptions with the probability of selecting a particular content optionbeing skewed based on its ranking. For example, the system may implementa weighted random selection algorithm. In some embodiments, such as whenrankings are determined based on a score value, the score value may beused to determine the probability that a particular content option isselected. For example, the system may select a content option using alottery system wherein the scores are used for the number of ticketentries a particular content option has in the lottery. In this manner,the score for each content option is directly related to the probabilitythat the content option is selected. In some examples, to further skewthe selection of content to the higher ranked content options,additional bonus score values may be provided to the higher rankedcontent options, such as the top three content options may receive anadditional 50% score value based on its own score. In this manner, asystem using the lottery method above would skew content selections moretowards the top three ranked content options. Although this exampleskews the selection to the top three, the skew can be to any number,such as top 10, top 5, and/or the like.

In some embodiments, on one or more of the content options may beeliminated based on the last time the content was used for and/or shownto the user or how closely matched the content option is with a recentlyused content by the system. In some examples, content with similarkeywords or that have keywords that overlap over a predetermined amountmay be considered as related. In some examples, closely matched contentmay be based on how the content is categorized. For example, in thecontext of the content being an advertisement, advertisements may becategorized based on subjects and advertisements similarly categorizedor under the same category with another advertisement recently used, orhaving a time stamp within a predetermined time, may be disregarded bythe system. In some examples, the system may remove key words associatedwith advertisements recently shown for ranking or determining whichadvertisement to select.

Additionally, in some embodiments, the system may account for usercontent preferences when selecting content options. For example, thesystem may have received feedback regarding one or more content optionsthat were selected for the user or displayed to the user previously. Insome examples, the feedback may be the removal of keywords used by thesystem to select future content for the user. In some embodiments, thesystem may remove those keywords in response to the feedback receivedfrom the user.

In some embodiments, keywords may be associated with weighted valuesbased on user feedback. For example, the feedback may have been a userincreasing a bar on a slider which may indicated that the user wouldlike to receive more similar content. In response, the system mayincrease the weight or value of one or more keywords associated with thecontent. In this manner, when the keywords associated with the user aresummed during the content selection process, keywords with a higherweight and/or value will contribute more to the ranking of the content.Similarly, a user may reduce the relevance of a keyword and the systemmay, in response, lower the weighted value of the keyword. In someexamples, the keywords may have negative values to further deter theuser of content that include that keyword.

In some examples, keywords may be weighted and/or used depending onmultiple factors, such as user feedback, social media trends, and/or thelike. For example, a user may provide feedback reducing the importanceof a keyword. As a response, the system may apply one or more conditionsto the keyword before it is used to select a content option for theuser. For example, one of the conditions may be that the keyword istrending on one or more social media websites. Trending can be athreshold value associated with the keyword social media velocity(mentions per second), acceleration (the change in velocity per second),a combination thereof, and/or the like. The system may, depending on theuser feedback, change the threshold value of the social media trend forthe keyword depending on the feedback. For example, the system mayincrease the velocity value for a condition for a keyword in response tothe user providing negative feedback for the keyword. Similarly, thesystem may lower or eliminate the trend value condition for the keywordin response to the user providing positive feedback for the keyword.

In some examples, the feedback may have been to indicate that certainprovided content was undesirable. In some embodiments, the system maydetermine which keywords are associated with the content that the userindicated was undesirable and remove those keywords from considerationwhen selecting another content option for the user.

In some examples, keywords may have a hierarchy or a “stickiness”ranking that may affect how easily keywords are removed fromconsideration regarding the selection of content. For example, thesystem may remove a keyword from use when selecting content aftermultiple negative indicators when the keyword is categorized as broad orhas a higher stickiness rating. In contrast, the system may remove akeyword after a single negative indicator or with less negativeindicators when the keyword is categorized as more specific and/or has alower stickiness rating. The stickiness rating may be a predeterminedvalue and/or calculated value based on how the word is used, how oftenthe word is used in social media, and/or how many different contentobjects are associated with the word.

In some examples, the hierarchy, broadness, and/or stickiness rating fora keyword may depend on the number of content options that the keywordis associated with. In some embodiments, the rating may be based on thepercentage of content options that use the keyword. For example, if thekeyword is used in less than 5% of content options, a single negativeindicator may cause the system to remove the keyword from use whenselecting content for that user. Similarly, if the keyword is usedbetween 5% and 30% of content options, the system may remove the keywordfrom use with a user after two negative indicators. Additionally, forkeywords above 30%, the system may remove the keyword from use with auser (based on the user profile) after three negative indicators. Insome examples, when a keyword is used in over a certain percentage ofcontent options, the keyword may not be removed unless a userspecifically requests removal of that keyword from use when interactingwith a system provided feedback mechanism. In some examples, differentnegative indicators may be provided different weight, and instead ofusing the number of negative indicators for removing the use of akeyword, there may be a threshold weight. For example, a negativeindicator for a keyword may be a user indicating content associated withthe keyword was undesirable. However, that content may be associatedwith multiple keywords. Negative indicators received in this manner maybe given a lower weight, such as half a point. In some examples, anegative indicator may be that a user did not click or interact with thecontent. Such an indicator may have even less weight, such as a tenth ofa point. In contrast, a direct indicator, such as a user providingfeedback that a keyword should not be used when targeting content to theuser (e.g. actuating an [x] mark on a keyword), may have a much higherweight or a maximum possible weight. In some examples, the system mayremove the use of a keyword for selecting content options for a userwhen a user provides a direct indicator that a keyword should not beused. An example of a direct indicator may be user provided feedbackdirected towards the keyword, such as the user selecting the keyword forremoval.

In some examples, process 300 may include operation 304. At operation304, the system may select a feedback mechanism for implementing withthe selected content. In some examples, the feedback mechanism maydepend, at least in part, on the type of content. In some examples thefeedback mechanism may depend at least in part on information about thedevice, such as whether the device is a mobile device. For example, formobile devices, additional feedback mechanisms may be available, such asfeedback mechanisms that implement facial recognition, touch screen,and/or audio based feedback. In some examples, the feedback mechanismchoice may depend at least in part on the type of feedback that wasreceived previously for the same content. In some examples, the feedbackmechanism may be used to obtain feedback to determine which keywords toassociate with a user, user profile, user device, content, and/or thelike, as described above in operation 303.

In some examples, the feedback mechanism may depend on information aboutthe user, and/or previous feedback received from the user, which may beinformation held in the user profile. The system may balance one or morefactors when attempting to select a particular feedback mechanism. Insome embodiments, the system may balance between the likelihood ofreceiving user engagement and the quality or amount of information thatthe feedback mechanism is obtaining or requesting. For example, havingan actuatable element that closes content like an x out box or “[x]” issimple and may obtain more engagement than a survey with data entryfields or a questionnaire with several multiple choice questions.However, when a user engages with a questionnaire or survey, the systemreceives more information and the information is of higher quality.

In some examples, the system may have predetermined, user provided,and/or admin provided values associated with the type of feedback. Forexample, a “[x]” type feedback may be valued at 1 and a response to asurvey may be valued at 100. Over time, as different feedback mechanismsare provided to a user and the system receives or does not receive thefeedback, the system can determine a rate of engagement for thatparticular user, such as 1 engagement over 1000 impressions. The valueof the particular feedback mechanism may be multiplied to the engagementrate to calculate an expected data value when selecting the feedbackmechanism. For example, if the “[x]” feedback mechanism had a feedbackrate of 499 engagements per 1000 views (499/1000) and the feedback valueof “[x]” was 1, the expected data value would be 499/1000. Similarly, ifthe rate of response to a survey was 9/1000 and the survey value was100, the expected data value for the survey would be 900/1000. In thisexample, the expected data value of providing the survey feedback ishigher than the “[x]” feedback, and the system may, in response, chooseto use the survey feedback mechanism. In some examples, the system mayrandomly select from the feedback mechanism options with the feedbackmechanism that has the higher expected value having an increased weightand likelihood of being chosen. In this manner, other forms of feedbackmechanisms may continue to be used and the system could adjust as thefeedback preferences of a user changes over time. One example of aweighted random selection would be to use a lottery system wherein thenumber of tickets to the lottery depends on the expected value of thefeedback mechanism. In some examples, the system may have a base lowerlimit of selection probability for a feedback mechanism to ensure thatthe lesser effective feedback mechanisms are still used regularly.

In some examples, instead of individualizing the feedback mechanismvalue calculation to a particular user, the system may conduct the valuecalculation for the specific content option or content category over aplurality of users. For example, the system may determine that for aparticular content option, the “[x]” feedback mechanism had a feedbackrate of 499/1000 over a plurality of users and the survey feedbackmechanism had a feedback rate of 9/1000 over a plurality of users. Thesystem may then select which is the best feedback mechanism for thatparticular content option in a similar manner as discussed above, suchas by lottery discussed above or in another manner.

In some examples, process 300 may include operation 305. At operation305, the system may provide the content selected at process 303 with thefeedback mechanism selected at operation 304 for display on the userdevice. In some examples, the system may provide the content andfeedback mechanism to a third party to display through their own webpageand/or application. In some examples, the system may cause the userdevice to display the content through an interface plugin, such as a webinterface, associated with the system.

In some examples, process 300 may include operation 306. At operation306, the system may determine whether a user provided feedback for thecontent. In some examples, the system may receive an indicator as towhether the user provided any feedback for the content. For example, athird-party system may monitor whether the user scrolled passed thecontent, moved to another flow of a webpage or application, left thewebpage or application displaying the content, paused the content, orother action that causes the content to no longer be displayed. Inresponse, the third-party system may provide an indicator to the systemthat the content was ignored. Similarly, if the user interacts with theadvertisements, such as clicking on the content, hovering a virtualpointer (e.g. mouse pointer) over the content, closing the content,interacting with the feedback mechanism, and/or the like, thethird-party may notify the system of the interaction, informationassociated with the interaction, and/or the interaction type. In someexamples, the system, rather than a third-party system, may monitor forinteractions with the content and/or feedback mechanism. Depending onwhether or not the system determines that feedback was received, thesystem may conduct one or more operations.

In some examples, process 300 may include operation 307A. In someexamples the system may conduct operation 307A in response todetermining that user feedback was not provided for the content atoperation 306. At operation 307A, the system may update the user profilewith regards to the feedback mechanism. For example, the system mayupdate in the user profile how many times the feedback mechanismselected at operation 305 has been used with this user and how manytimes it has been successful and/or unsuccessful at obtaining userengagement and/or feedback. In this manner, feedback selection operation304 is updated for when the next feedback mechanism is to be selected.For example, the likelihood that the feedback mechanism used atoperation 305 is reduced for the particular user and/or user profile.

In some examples, process 300 may include operation 307B. In someexamples the system may conduct operation 307B in response todetermining that feedback was received at operation 306. At operation307B, the system may update the user profile on how content is selectedfor this user based on the feedback. For example, the user may haveselected a keyword indicating that the user is uninterested in contentassociated with that keyword. In response, the system may update theuser profile to eliminate the keyword, reduce the weight of the keyword,and/or take other actions as discussed above. Furthermore, the systemmay track which feedback mechanism was used with the content such thatthe same feedback mechanism is not used again with the same content.Furthermore, statistics regarding which feedback mechanism was used maybe updated for the system to select the next feedback mechanism for theuser at operation 304.

In some examples, process 300 may include operation 308. At operation308, the system may update information about the content used to bettertarget the content. For example, if no feedback was received for thecontent, the system may note that the content did not receive anyfeedback. In contrast, if feedback was received, the system may keeptrack of which feedback mechanism was used. In this manner the systemmay use this information to help select which feedback mechanism to useat operation 304 with this content the next time this content isselected at operation 303.

FIG. 4 illustrates an example graphical user interface for a feedbackmechanism 400 where the user can select keywords, according to someembodiments. In some examples, feedback mechanism 400 may be one of thefeedback mechanisms that may be selected at operation 304 of FIG. 3.

As shown, feedback mechanism 400 is a graphical user interface that maybe displayed on a user device, such as a smart phone, tablet computer,personal computer, and/or other computing devices. In some examplesfeedback mechanism 400 may be overlaid over content, such asadvertisement 410. In some examples, as shown, advertisement 410 maydisplay an image associated with the product and/or service beingadvertised. In some examples, advertisement 410 may be an actuatableelement, such as a hyperlink, that may direct a user to a website and/orapplication associated with advertisement 410.

Feedback mechanism 400 may be shaded or colored in a manner such that auser can easily distinguish the feedback mechanism 400 fromadvertisement 410. Furthermore, feedback mechanism 400 may be partiallytransparent such that advertisement 410 is still viewable behindfeedback mechanism 400 when overlaid over advertisement 410. The shadedportion of feedback mechanism 400 that overlays advertisement 410 may benot be actuatable and may also cause that portion of the advertisementnot to be actuatable due to feedback mechanism 400. In this manner, auser does not have to fear that they may accidentally click on theadvertisement or content when giving feedback.

In some examples, feedback mechanism 400 may display one or morekeywords 420 associated with advertisement 410. In some examples,keywords 420 may be provided by a system such as one or more systems ofFIG. 1 and/or a system implementing process 300 of FIG. 3. In someexamples, keywords 420 may be actuatable elements that a user canselect. In some examples, the “[x]” portion of keywords 420 may be theactuatable elements. A user may click or select one or more of keywords420 to indicate to the system that the user would prefer advertisementsthat were not associated with those keywords.

FIG. 5 illustrates an example graphical user interface of feedbackmechanism 500 where the user can adjust the weight of keywords using aslider, according to some embodiments. In some examples, feedbackmechanism 500 may be one of the feedback mechanisms that may be selectedat operation 304 of FIG. 3. As shown, feedback mechanism 500 is agraphical user interface that may be displayed on a user device, such asa smart phone, tablet computer, personal computer, and/or othercomputing devices.

Similar to feedback mechanism 400 of FIG. 4, feedback mechanism 500 maybe shaded or colored in a manner such that a user can easily distinguishfeedback mechanism 500 from content such as news article 510.Furthermore, feedback mechanism 500 may also be partially transparentsuch that news article 510 is still viewable behind feedback mechanism500 when overlaid on top of news article 510. The shaded portion offeedback mechanism 500 that overlays news article 510 may not beactuatable and may also cause that portion of the article not to beactuatable due to feedback mechanism 500. In this manner, the feedbackmechanism does not punish a user for an accident miss click on a blankportion of feedback mechanism 500.

Similar to feedback mechanism 400 of FIG. 4, feedback mechanism 500 maydisplay keywords 520 associated with news article 510 being displayed.Keywords 520 may be one or more keywords that were used in deciding toimplement news article 510. In some examples, keywords 520 may be partof a slider GUI input element with sliders 530. In some examples,sliders 530 may be placed initially at current user settings associatedwith the keywords. The placement of sliders 530 may indicate theimportance of keywords 520 for advertisement targeting to the user, therelative weight value of keywords 520 associated with the user, and/orthe like.

In some examples, the user may select and move one or more sliders 530to send feedback to the system that selected the advertisement. In thismanner, the advertisement selecting system may adjust user preferencessuch that the advertisement selection system may better selectadvertisements relevant to the user. Additionally, this feedback couldbe used by the advertisement selecting system to gather metrics for theadvertisements such that advertisers may better target and/or createadvertisements.

In some examples, such as when news article 510 is for news articles,sliders 530 may represent popularity of an article before targeting itto the user. For example, one end of the slider may indicate that theuser is interested in every article associated with the keyword,independent of how important the system determines the article is. Asecond end of the slider may indicate that articles associated with thekeyword should not be shown to the user or not be shown unless thesystem determines that the article is important, trending, and/orpopular. In some examples different sliders may represent differentsystem preferences and the system may provide indicators as to what thesliders control (not shown). Furthermore, in some examples, each keywordmay have multiple sliders for indicating preferences such as importance,relevance to the advertised product, relevance to the user, and/or thelike. For example, many keywords may be used in different context andthe user may be able to use the slider to better train a content systemto correctly determine the context in which the user is interested infor a word. For example, a user may be interested in football as theword is used internationally but not for American football. The use mayindicate using the slider that the word football as used in articleabout American football was not contextually correct but have anotherslider that indicates that football is important to the user. In thismanner, the advertisement selection system may be able to determine thatthe user is interested in football in the context of soccer but notAmerican football.

The foregoing disclosure is not intended to limit the present disclosureto the precise forms or particular fields of use disclosed. As such, itis contemplated that various alternate embodiments and/or modificationsto the present disclosure, whether explicitly described or impliedherein, are possible in light of the disclosure. For example, some ofthe above embodiments have focused on advertisements and news articles,however, the system may work with other content. For example, the systemmay work for recommending video content, virtual reality content,augmented reality content, audio content and/or the like. Having thusdescribed embodiments of the present disclosure, persons of ordinaryskill in the art will recognize that changes may be made in form anddetail without departing from the scope of the present disclosure. Thus,the present disclosure is limited only by the claims.

What is claimed is:
 1. A system, comprising: a non-transitory memorystoring instructions; and one or more hardware processors coupled to thenon-transitory memory and configured to read the instructions from thenon-transitory memory to cause the system to perform operationscomprising: in response to receiving a content request, retrieving auser profile associated with a user; determining content for the userbased on user preferences included in the user profile; extracting oneor more keywords from the content; generating a feedback interfaceincluding a selectable feedback mechanism for receiving feedbackcorresponding to the one or more keywords based on user feedbackpreferences included in the user profile; and causing a user deviceassociated with the user profile to display the feedback interface andthe content.
 2. The system of claim 1, wherein the operations furthercomprising: receiving user feedback data based on an interaction of theuser with the feedback interface or a lack of user interaction with thefeedback interface; and adjusting the user feedback preferences of theuser profile based on the user feedback data.
 3. The system of claim 1,wherein the operations further comprise selecting, from a plurality ofselectable feedback mechanisms, the selectable feedback mechanism forgenerating the feedback interface.
 4. The system of claim 3, wherein theoperations further comprise: determining a set of user input componentsavailable on the user device, wherein the selectable feedback mechanismis selected from the plurality of selectable feedback mechanisms basedon the set of user input components available on the user device.
 5. Thesystem of claim 3, wherein the selectable feedback mechanism is selectedfrom the plurality of selectable feedback mechanisms user feedbackpreferences included in the user profile.
 6. The system of claim 2,wherein the adjusting the user feedback preferences comprises removingone or more labels from the user profile based on a negative indicatorcorresponding to a keyword of the one or more keywords.
 7. The system ofclaim 6, wherein the negative indicator indicates that the user did notinteract with the content or that the user has provided a negativefeedback using the selectable feedback mechanism.
 8. A methodcomprising: obtaining, by one or more hardware processors, a userprofile associated with a user, wherein the user profile comprisescontent preferences and feedback preferences of the user; determining,by the one or more hardware processors, based at least in part on thecontent preferences of the user profile associated with a user, firstelectronic content for the user; extracting, by the one or more hardwareprocessors, one or more keywords from the first electronic content;generating, by the one or more hardware processors, a feedback interfaceaccording to a first feedback mechanism selected for the user, whereinthe feedback interface is configured to obtain a feedback inputcorresponding to the one or more keywords from the user according to afirst user input interface type specified by the first feedbackmechanism; incorporating, by the one or more hardware processors, thefeedback interface into the first electronic content to generatemodified first electronic content; and displaying, by the one or morehardware processors on a user device associated with the user, themodified first electronic content.
 9. The method of claim 8, furthercomprising: detecting an interaction of the user with the feedbackinterface; and adjusting the feedback preferences of the user profilebased on the detected interaction.
 10. The method of claim 8, furthercomprising selecting, from a plurality of electronic feedbackmechanisms, the first feedback mechanism based on the feedbackpreferences of the user profile.
 11. The method of claim 10, furthercomprising: determining, from a plurality of user input components, thata set of user input components are available on the user device; whereinthe first feedback mechanism is selected from the plurality ofelectronic feedback mechanisms further based on the set of user inputcomponents available on the user device.
 12. The method of claim 9,further comprising: subsequent to displaying the modified firstelectronic content on the user device, determining second electroniccontent for the user; selecting, from the plurality of feedbackmechanisms, a second feedback mechanism based on the adjusted feedbackpreferences; generating a second feedback interface according to thesecond feedback mechanism; and displaying the second feedback interfaceand the second electronic content on the user device.
 13. The method ofclaim 10, wherein the first feedback mechanism is selected from theplurality of electronic feedback mechanisms further based on the firstelectronic content determined for the user.
 14. A non-transitorymachine-readable medium having stored thereon machine-readableinstructions executable to cause a machine to perform operationscomprising: retrieving a user profile associated with a user;determining content for the user based on content preferences of theuser profile; extracting one or more keywords from the content;generating a feedback interface including a selectable feedbackmechanism for determining feedback data corresponding to the one or morekeywords, wherein the selectable feedback mechanism is selected based onuser feedback preferences included in the user profile; and causing auser device associated with the user profile to display the feedbackinterface and the content.
 15. The non-transitory machine-readablemedium of claim 14, wherein the operations further comprise: determiningthe feedback data through the feedback interface; and adjusting the userfeedback preferences of the user profile based on the feedback data. 16.The non-transitory machine-readable medium of claim 15, wherein the userfeedback data is determined based on an interaction of the user with thefeedback interface or a lack of interaction with the feedback interface.17. The non-transitory machine-readable medium of claim 14, wherein theoperations further comprise: determining a set of user input componentsavailable on the user device; and selecting, from a plurality ofselectable feedback mechanisms, the selectable feedback mechanism forgenerating the feedback interface based on the set of user inputcomponents available on the user device.
 18. The non-transitorymachine-readable medium of claim 14, wherein the selectable feedbackmechanism is selected from the plurality of selectable feedbackmechanisms further based on the content determined for the user.
 19. Thenon-transitory machine-readable medium of claim 15, wherein theadjusting the user feedback preferences comprises removing labels fromthe user profile based on a negative indicator.
 20. The non-transitorymachine-readable medium of claim 19, wherein the negative indicatorindicates that the user did not interact with the content or that theuser has provided a negative feedback via the feedback interface.